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Abstract. Does STEM education improve students’ higher-level thinking 
and cognitive abilities? So far, empirical research has not yielded con- 
sistent conclusions. As such, this study applied the method of meta- 
analysis to synthesize quantitatively existing research to better under- 
stand STEM and its effects on students’ abilities associated with learn- 
ing. The study found that STEM education is conducive to improving 
students' higher-order thinking and cognitive ability levels with an effect 
size of (d = 0.798). The results indicate that teaching methods and stu- 
dent experiences in STEM education have a positive effect on student 
learning. 
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INCE the 1980’s, when the National Research Council (NRC) advocated 

strengthening undergraduate sciences, mathematics, engineering, and technology 

education, STEM education was implemented in the United States. It has been 
gradually expanded from higher education to K-12 education. During this transition, 
various departments of the federal government issued large numbers of policies and 
reports to increase STEM education and putting in place financial inputs to ensure im- 
plementation. One of the most prominent reasons for doing so was to maintain the Unit- 
ed States prominent roles in world politics and economics (Thomasian, 2007; CoSTEM, 
2013; Honey et al., 2014). The realization of the strategic significance of STEM educa- 
tion had a number of other countries around the world also begin experimenting with 
STEM education (Marginson et al., 2013). These actions have led researchers, primarily 
in education, to analyze whether STEM education is conducive to improving students’ 
learning abilities. 

To this point, STEM education evaluation has mainly focused on the outcomes of 
students. As far as the content of the evaluation is concerned, it includes not only the 
students' academic achievements, but also, measurements of their abilities and tenden- 
cies towards subject learning and employment. For instance, a general assessment sys- 
tem built by the STEM Education Association in Portland, Oregon, required the meas- 
uring of students’ use of knowledge, higher-level cognitive abilities, academic ac- 
ceptance, and motivational resilience (Saxton et al., 2014). The National Research 
Council also reported that it was not sufficient to only be concerned with students’ test 
scores when evaluating STEM education, but to also consider the impact on students’ 
interests, creativity and behavior. However, there is no agreement on whether and how 
much STEM education affects students’ abilities to learn. Yildirim (2016) systematical- 
ly analyzed the improvements of students’ innovative abilities through STEM education; 
but, failed to specify to what extent are its affects. Sarac (2018) posited that STEM edu- 
cation can improve students’ scientific process skills (d = 0.820); yet, did not answer 
whether STEM education can help to improve the skills required by STEM profession- 
als in the broader context. In contrast to the relatively optimistic findings of the afore- 
mentioned, Jang (2016) pointed out that STEM education did not assist the 18 im- 
portant abilities of STEM professionals, and therefore, a more cautious approach to the 
effectiveness of STEM education needed to be taken. 


Literature Review and Theoretical Framework 
STEM Education 


Research has not yet formed a unified understanding of STEM education. Carmichael 
(2017) analyzed the policy texts of various states in the United States and found that the 
states, for the most part, had different understandings of STEM education and how it 
was to be implemented. Hence, defining the operations of STEM education became a 
primary objective. 
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The STEM Education Act of 2015, defined STEM education as “education in sci- 
ence, technology, engineering, mathematics and other fields, including education in 
computer science” (US Congress, 2015). The stringent focus on subject areas failed to 
make a comprehensive summary of the rich connotation of STEM education (Sanders, 
2009. 

We believe that a more reasonable and comprehensive definition of STEM educa- 
tion is needed to better reflect the developmental process and full characteristics of the 
program. In its early stages, STEM education focused on the knowledge of subjects and 
ignored the links (Atkinson & Mayo, 2010), which to some degree weakened students’ 
interest in STEM subjects and lowered their academic performance (Kelley & Knowles, 
2016). For this reason, the United States has proposed further reforms, including the 
implementation of STEM integrated education, thus to enhance students’ interest and 
ultimately improving their learning achievements (Honey, et al., 2014). This integration 
of the disciplines expanded the effectiveness by way of integrating them to real-world 
situations and problems, using problem-based, inquiry-based, and project-based learn- 
ing. 

From the perspective of STEM education, we consider that STEM education has 
two outstanding characteristics: first, it emphasizes the integration of science, technolo- 
gy, engineering and mathematics; and, secondly transforms traditional teaching models 
to a student-center model. As defined by the National Association of High Schools, 
STEM education is one that breaks the boundaries of traditional subjects and integrates 
teaching and learning of science, technology, engineering and mathematics as a guide 
that encourages students to solve problems using their newly learned knowledge. All in 
all, we defined STEM education as: an education approach which based on authentic 
environment and integrate science, technology, engineering and mathematics or more 
other subjects by students-centered learning model, such as project-based learning, de- 
sign-based learning, inquiry-based learning and so on, to cultivate students’ ability and 
improve their achievement. 


Higher-order Thinking and Higher-order Cognitive Abilities 


Bloom et al. (1956), classified educational goals as to being: knowledge, comprehen- 
sion, application, analysis, synthesis and evaluation. Many scholars regard knowledge 
as low-order thinking and of low cognitive ability; whereas, the other remaining skills 
from Bloom’s Taxonomy as being higher-order thinking and of higher cognitive abili- 
ties (Miri, 2007). Wood (2007) divided higher-level cognitive abilities into three dimen- 
sions: problem-solving, evidence-based discussion, and metacognitive. Considering the 
prominent purpose of education to develop skills more than knowledge, scholars have 
explored the influence of different teaching methods. Hemlo & Ferrari (1997) examined 
how to cultivate students’ higher-order thinking based on problem-based learning. Hop- 
son (2001) found that the application of educational technology in classroom teaching 
can also improve students’ higher thinking abilities. Lastly, Zohar and Dori (2003) dis- 
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covered that after attending training projects using of high-order thinking abilities, poor 
performing students had a larger net increase compared to higher performing students. 


The Relationship between STEM Education and Students’ Abilities 


In existing empirical literature, results on the relationship between STEM education and 
students’ abilities are inconsistent. Some studies indicate that STEM education can sig- 
nificantly improve students’ abilities levels and there is a large effect size. For example, 
Fan and Yu (2017) found that engineering-based technology education showed large 
improvement (7? = 0.18) in high school students’ higher-order thinking. Li et al (2016) 
discovered that when comparing with non-STEM education, STEM education signifi- 
cantly improved students' problem-solving abilities (d = 0.526). However, other studies 
had noticed STEM education, though beneficial to students’, was much less effective. 
Cakir et al (2016) found that STEM education had improved the level of students’ re- 
flective thinking abilities, but at a rate of (d = 0. 1319). Psycharis & Kallia (2017) study 
on computer programming-based learning saw only a small influence on students’ criti- 
cal thinking (d = 0.229). Then, there were studies that observed no effects on students’ 
abilities levels, such as, Choi and Hong (2015) (creative problem-solving) and Anwari 
(2015) (metacognition). 

Further studies realized that the effectiveness of STEM education can be influ- 
enced by external factors. Inman (2011) discovered that STEM education can improve 
students’ scientific inquiry abilities; yet, the degree of effectiveness was influenced by 
socio-economic factors. Taylor (2016) research showed that students' experience in 
STEM education would affect the improvement of problem-solving ability, and the ef- 
fect of STEM education is better for novice students. 

Based on the above review, this study will focus on answering the following three 
questions through meta-analysis: 

1) Is STEM education conducive to improving students’ abilities? 

2) To what extent does STEM education affect students’ abilities? 

3) During the process of STEM education, what factors will be influenced? 


Research Method 


The method used for the study was meta-analysis which is a quantitative synthesis 
method to review literature. Initially, this method was applied to synthetically evaluate 
results of clinical psychology research. Compared with traditional methods, which often 
relied on subjective judgments, meta-analysis is objective, systematic, and evidence- 
based. Thus, gaining acceptance in the fields makes up the social sciences (Lipsey & 
Wilson, 2011). 

On the whole, the main steps of meta-analysis are: 1) enacting inclusion criteria; 2) 
searching and filtering documents; 3) coding documents and extracting data; 4) as- 
sessing the quality of the included documents; 5) calculating the mean effect size; 6) 
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analyzing heterogeneity; and, 7) testing publication bias and sensitivity. These steps 
were followed to answer the research questions posed in this study. 


Inclusion Criteria 


We collected and screened selected literature according to the following inclusion crite- 
ria: 

(a) The literature enrolled was published in English between 1996 and 2018 and 
the type of literatures was not limited. After all, English is an internationally accepted 
academic language and most of the research on the effects of STEM education is pub- 
lished in this language. 

(b) The content focuses on STEM education and students’ abilities in elementary 
education. As well, the impact of STEM education on the abilities of students’ in ele- 
mentary education, excluding special education, vocational education, and after school 
programs. 

(c) Literature that included comparisons between STEM education and non-STEM 
education using effect size. Evaluations were made using the criteria of Cheung and 
Slavin (2013a) — large differences in pretest were excluded (ES > 0.5) and randomized 
trials without pre-test are included. 

(d) At least two teachers were teaching the experimental group and the control 
group had to be separated to minimize the impact of teacher factors. This was done be- 
cause if the two groups were taught by only one teacher, the independence of the inter- 
vention could not be guaranteed. 

(e) To avoid possible deviation of experimental results, students were not alerted 
of the reason for doing specific functions. 

(f) The sample size of the experimental group and control group were similar. This 
was done to avoid bias. 

(g) The statistical information needed to be sufficient, so that, effect size could be 
calculated. 


Searching and Scaffolding Documents 


The key terms used during Google Scholar and ERIC searches included: STEM educa- 
tion, higher-order cognitive skills, higher-order thinking, creative skills, innovation ca- 
pacity, creativity, problem-solving skills, problem-solving ability, ability, skill. In all, 
28012 studies were found and designated for this study. A further breakdown is shown 
in Figure 1. 


Coding and Data Extraction 


In heterogeneity analysis, the literature should be grouped according to research design 
and sample characteristics. Therefore, coding was done accordingly: 

(a) Gender (Ge): female code-0, male code-1. Samples containing male and female 
was reported as 2 and unreported samples as null. 
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Figure1. Search and Screening Process. 
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(b) Family socioeconomic background (SES), Low SES coded-l, medium SES 
coded-m, high SES coded-h. A sample containing low, medium and high SES was cod- 


ed-mix, and unreported samples coded-null. 

(c) Race (E), White coded-w, Afro-American coded-b, Asian coded-a, Hispanic 
coded-h, and others code-o. A sample contains multiple ethnic groups code-mix, and for 
non-reported code-null. 

(d) Grade (Gr): K-5 code-P, grade 6-8 code-m, grade 9-12 code-h. 

(e) Ability type (AT) 

(f) Area (Lo) 

(g) STEM Teaching Method (TA), Project-based Learning coded-PBL, Problem- 
based Learning (PBL) coded-pbl, Inquiry-based Learning coded-IBL, Design-based 
Learning coded-DBL, and other STEM educational measures were coded-other. 

(h) Research design (Rd), the quasi experimental design coded-QE, and the ran- 
dom experimental design was coded-re. 
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(i) Duration of intervention (D), According to the length of the experiment, it was 
divided into four levels: 0-2 months, 2-4 months, 4-6 months, 6 months+. 

(j) Sample size (Ss), using the guidelines set out by Cheung & Slavin (2013b), 
studies with a sample size greater than 250 were designated as large sample studies, and 
studies with a sample size less than 250 were assigned as small studies. Accordingly, 
we coded the large sample studies as | and the small sample studies as s. (Table 1) 


Table 1. Details of Included Resear 


Study Ge SES E Gr TA OD AT Lo Ss 
Childress, 1996 Null Null Null M Other 02 — Frablemsolving  Us_ small 
Eseryel, 2011. 2 Null Null M — Other — ao USA _ Large 
Lartson, 2018 2 9 Mix «Mix =H DBL “4 = FrablemsoWing ga Small 
crae ae 5 Null Null Null mM peL  ** Anal Kenya Small 
Rehmat,2015 2 Null Mix P Pbl “© — Griticalthinking © USA Small 
Robinson. et aly Niu Null Null P  IBL = & Science process sq small 
SO Null rNal SING PEN IGIy oe Aa ce ice Pecos. Sek Sina 
sorhishetals 2 Null Null P BL © = Seencaprocess sq Large 
Ss hc 2 Null Null H Other °%  Gritical thinking © USA Small 
haps 2 Null Null H Other 2 — Reasoningskills USA Small 
Hashem, 2015a 2 Mix Mix M Other ** — Griticalthinking © USA Small 
Hashem, 20156 2 Mix Mix M Other ** — Griticalthinking © USA Small 


*: The author does not specify which ability is tested in the higher-order cognitive ability dimension, but is 
generally referred to as "higher-order cognitive skills". 


Quality Assessment of Studies 


Considering that meta-analysis is a method of quantitative synthesis of existing research, 
the quality of the included literature will affect the quality of the final results. Referring 
to Valentine & Cooper's (2003) method for evaluating the quality of literature, this 
study assessed the quality from five aspects: whether the literature clearly described the 
interventions, research design, sample characteristics, testing tools and measurement 
processes (unclear-1, somewhat clear-2 and clear-3). As Table 2 shows, the quality of 
the literature included was high enough to meet the needs of this study. 


Vol.1, No.1, 2018 9 


Zeng et al. STEM on Student’s Ability 


Table 2. Document Grades. 


Research Sample Charac- Test Measure 


eiudy Imervention Design teristics Tool Process Total 
Childress, 1996 2 2 1 1 2 

Eseryel, 2011 2 2 1 2 2 

Lartson, 2013 3 3 3 2 2 13 
Kibett, & Kathuri, 

2015 2 2 1 2 2 9 
Rehmat, 2015 3 2 2 2 2 11 
Robinson, et al., 

2014a 3 2 1 2 3 11 
Robinson, et al., 

2014b 3 2 1 2 3 11 
Cotabish, et al., 

2013 2 2 2 2 2 10 
Psycharis, & 

Kallia, 20174 a 2 : e : © 
Psycharis, & 

Kallia, 20176 a 7 : . 4 S 
Hashem, 2015a 3 2 3 2 3 13 
Hashem, 2015b 3 2 3 2 3 13 


Analysis of Result and Discussion 
Combining Effects 


This study used Comprehensive Meta-Analysis Vision 2 software to calculate the ef- 
fects. From the selected samples, it cannot be assumed that the reported effects of all 
documents were consistent and the results of the heterogeneity reported a significance 
of (Q = 58.950, p < 0.0001). Therefore, a random-effect model (Michael Borenstein et 
al., 2009, p83) was used for further analysis. Used was the “one study removed” meth- 
od to exclude possible outliers. The principle behind this method was to enable calcula- 
tion of the average effect of the documents. If the deviation between the calculated av- 
erage effect and the original value was too large, the documents were deemed as ab- 
normal values and were not included in the final effect analysis (Young, et al., 2017. As 
shown in Table 3, the effects of the studies were distributed between 0.229 and 1.647. 
Eight of the effects were statistically significant and the remaining three were not sig- 
nificant. The combined effect d = 0.798 (p < 0.0001) in the random effect model was 
calculated, which was a moderate effect according to Cohen’s (1988) criteria. To a cer- 
tain extent, this result can answer questions | and 2, STEM education is conducive to 
improving students’ higher-order cognitive abilities and higher-order thinking abilities. 
It also shows that STEM education can cultivate students’ ability to meet the needs of 
the STEM labor market. 
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Table 3. Effect and Combined Effect Volume. 


Statistics for Each Study 
Study Ability c U 7 
, ‘ ower pper - 7 
Cohens’d SE Variance limit limit wala p-value 
Childress, Problem- 
1996 solving skills 0.551 0.355 0.126 -0.145 1.246 1.552 0.121 
Eseryel, Problem- 
5011 solving skills 0.303 0.131 0.017 0.047 0.559 2.316 0.021 
Lartson, Problem- 
5013 solving skills 1.015 0.248 0.062 0.528 1.502 4.088  <0.0001 
Kibett, & Higher-order 
Kathuri, cognitive 1.647 0.186 0.035 1.283 2.011 8.865 <0.0001 
2015 skills 
Robinson, Science 
et al., process 1.437 0.276 0.076 0.897 1.978 5.211 <0.0001 
2014a skills 
Robinson, Science 
et al., process 0.585 0.191 0.037 0.209 0.960 3.054 0.002 
2014b skills 
d Science 
scent et process 0.497 0.077 0.006 0.346 0.649 6.430 <0.0001 
= skills 
Eeyehais, Critical 
& Kallia, cate 0.229 0.254 0.064 -0.268 0.726 0.902 0.367 
thinking 
2017a 
Psycharis, ' 
& Kallia, Fea Geen 0.260 0.067 0.007 1.011 1.933 0.053 
skills 
2017b 
Hashem, Critical 
2015a thinking 0.800 0.235 0.055 0.340 1.260 3.408 0.001 
Hashem, Critical 
2015b thinking 1.408 0.331 0.109 0.759 2.056 4.255 <0.0001 
Mean ES 0.798 0.143 0.021 0.517 1.079 5.568 <0.0001 


Analysis of Heterogeneity 


We used moderator analysis (Yong et al., 2017) to explore the source of heterogeneity. 
Due to the insufficient sample characteristics reported in the literature, this study was 
unable to treat gender, SES and ethnic variables as moderators and, at the same time, 
Kibett & Kathuri (2005) study did not report the items under higher-order cognitive 
skills, so ability type was excluded. Consequently, grade level, STEM education ap- 
proaches, experi mental duration and sample size were chosen as moderators to test 
heterogeneity. Test results (see Table 4) showed STEM education approaches as the 
leading factor of heterogeneity (Qg = 39.101, p < 0.0001), meaning different STEM 
education approaches had different effects on students’ abilities. Teaching approaches 
and project-based learning had the best effects, while problem-based learning had no 
effect on the improvement of students’ abilities. There was only one study dealing with 
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Table 4. Analysis of Moderators Effect Size. 


Moderator K Qs ES 95% Cl p-value 
a 3 0.568 0.432 0.705 <0.0001 
Grade level Nisgle 5 4.026 (p=0.134) 9.793 0.616 0.970 <0.0001 
school 
High school 3 0.589 0.302 0.877 <0.0001 
DBL 1 1.015 0.528 1.502 <0.0001 
STEM wducationap: IBL 3 39401 0.568 0.432 0.705 <0.0001 
proach Other 4 (p<0.0001) 0.342 0.143 0.541 0.001 
PBL 3 1.335 1.074 1.596 <0.0001 
0-2 mo 3 0.401 0.085 0.718 0.013 
Duration 2-4 mo 5 8.024 (p=0.017) 0.835 0.663 1.006 <0.0001 
6 mo+ 3 0.568 0.432 0.705 <0.0001 
Large 2 91774 0.447 0.317 0.578 <0.0001 
Sample size 
Small g  (p<0.0001) 0.938 0.778 1.097 <0.0001 


problem-based learning and the result was that it had no effect on improving students’ 
abilities (this area needs further study). Moderator’s experimental duration and sample 
size indicated influence on the mean effect size (Qg = 8.024, p = 0.017). As for experi- 
mental duration, STEM education is more efficient during 2-4 months, longer or shorter 
intervals had suboptimal results. Robinson and his colleagues (2014) found students 
during the first year of STEM education perform well in the science process skill tests 
but not as well in the second year. Taylor (2016), meanwhile, found that effects of 
STEM education can be influenced by students’ learning experiences. Novice learners 
gained more learning than those students considered higher achievers. It appears as the 
STEM program goes on, the impact of STEM education on student abilities gradually 
declines. Grade level was the only factor that had no significant influence on the mean 
effect size, which means that STEM education is suitable for all K-12 students. 

In regard to research question 3 of this study: it was found that STEM education 
approaches and students’ learning experiences are the moderator variables on students’ 
abilities. The results, though, could not identify whether the demographic factors and 
ability types had any impact on the mean effect size of STEM education. 


Publication Biases and Sensitivity Testing 
In the meta-analysis process, a funnel plot is commonly used to test for publication bias. 


If there is publication bias, it will lead to skewed final result and the calculated mean 
effect size would need to be treated with caution. The publication bias test for this study 
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is shown in Figure 2; results show a funnel plot that is basically symmetrical, meaning 
that there is no publication bias. 


0.0 


SE 


0.4 
0.0 0.6 1.0 1.6 


Standard Difference in Means 


Figure 2. Publication Bias Test (funnel figure). 


In order to judge the robustness of the analysis results, a classic fail-safe N test was 
used. This test was employed to calculate the minimum number of unpublished studies 
that could reverse the final results, in particular, in the area of robustness. A larger N 
means that the difference between the included studies and excluded studies would af- 
fect results more so; hence, the results of the meta-analysis would be more robust 
(Rosenthal, 1979). This study’s fail-safe N was 449 (p < 0.0001), which means we 
needed to include an additional 449 studies to attain robustness. 


Limitation of Research 


There are two limitations in this meta-analysis. First, the number of included studies is 
slightly insufficient. The main reason for this deficiency is that there are few empirical 
studies on the relationship between STEM education and students’ abilities, especially 
in the Asian region. Secondly, when studies were included, we limited the research de- 
sign to two experimental design groups. Though, the process of exploring for causal 
relationships between variables are the standard, this criterion would have declined the 
sample size of this study’s meta-analysis. 


Findings and Conclusions 
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The meta-analysis used to synthesize existing empirical research on the relationship 
between STEM education and K-12 students’ higher-order thinking and cognitive abili- 
ties found that STEM education is conducive to improving students’ ability levels. The 
mean effect size (d = 0.798, p < 0.0001) is large enough to support this conclusion us- 
ing Cohen (1988) principle. The results for heterogeneity analysis indicated that both 
STEM education processes and students’ learning experiences can influence the effects 
of STEM education. Lastly, there was no significant difference in STEM education ef- 
fect among students for different grades, indicating that STEM education is an effective 
model for all K-12 students’ development of higher-order abilities. 

Based on the research of this paper, it is concluded that there are still many gaps, 
yet, to be filled. First, researchers have supported the hypothesis that STEM education 
has more effect on students’ achievement than non-STEM education (Becker & Park, 
2011; Sarac, 2018), but it lacks enough studies to fully explain which STEM education 
practices would best fit for specific subjects and learning environments. Secondly, more 
research is needed to identify which factors influence the effects of STEM education in 
the area of human capital accumulation. For instance, groups such as women, African- 
Americans, Hispanics, and Asians are disproportionately underrepresented in current 
STEM education research (Beede et al., 2011; Koch et al., 2011; US Department of 
Education, 2016). 
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